There are different variants of Particle Swarm Optimization (PSO) algorithm such as Adaptive Particle Swarm Optimization (APSO) and Particle Swarm Optimization with an Aging Leader and Challengers (ALC-PSO). These algorithms improve the performance of PSO in terms of finding the best solution and accelerating the convergence speed. However, these algorithms are computationally intensive. The goal of this paper is high performance implementations of Traditional PSO (TPSO), APSO and ALCPSO using CUDA technology. We have implemented these three algorithms on both central Processing Unit (CPU) and Graphics Processing Unit (GPU) in order to analyze and improve their performance and reduce their computational times. We have achieved speedups up to 14. 5x, 31x, and 152x, for GPUTPSO, GPU-ALCPSO, and GPU-APSO, respectively. In addition, different number of threads has been chosen in order to find an appropriate number of threads per block for both APSO and ALC-PSO algorithms. Our experimental results show that the best choice for number of threads per block depends on the number of existing variables and constants in each algorithm and the number of registers per multiprocessor.